Method and Apparatus for Evaluating Integrated Circuit 
Packages Having Three Dimensional Features 

Claiming Priority 

^ Under the provisions oV^U.S.C. 1 19(e) and Patent Rule 55, Applicant 

5<5[ claims the benefit of the followin|-?Hor provisional application, Application No. 
60/257,763, filed October 5, 1999. 

Field of the Invention 
This invention relates to three dimensional sensing systems and methods 
for acquiring and evaluating images including data characteristic of three dimensional 
10 features on an object. 

Background of the Invention 

As integrated circuit packages increase in capability and complexity, the 
number of wires or contacts needed to interconnect the IC package to its associated 
substrate increases. Two of the methods used to increase the number of interconnects are 

1 5 to decrease the size and increase the density of wires around the sides of the IC package 
or form a two dimensional array of interconnects, or balls, on the underside of the IC 
package. Examples of these types of IC packages are the quad flat pack (QFP) which 
can have up to 600 wires on four sides of an IC package, or the ball grid array (BGA) 
which can have over 1000 solder balls that form interconnects on the bottom of the IC 

20 package. The wires or leads of a QFP and the balls of a BGA are three dimensional 
features. That is, the balls and wires of the various packages are project away from the 
package. 

The substrate or waferNto which the QFP or BGA attaches includes a 
corresponding array of pads to which theWres ("leads") of the QFP or balls of the BGA 
25 mate. It is of critical importance in the manufacture of electronics that the individual leads 
<yi or balls are coplanar with one another. If theMeads or balls are not coplanar and the IC 
C tV package is soldered onto a substrate some ofUlWeads or balls may be in electrical 
t contact with the substrate while others will not beYThe connections between a BGA and 
its substrate are not subject to visual inspection. Further, once a BGA is in place is often 
30 easier to discard the entire substrate with BGA if the\BGA is faulty. That is if a single 
IC package on a substrate that potentially contains nuJnerous IC packages s is not fully 



connected, then the entire jaiitarcrate is defective and must either be repaired or but in the 
case of BGAs is often diseased. 

The critical nature of attaching IC packages such as QFPs or BGAs to 
their substrates to have connectivity has created a need to measure the coplanarity of 
QFPs and BGAs. In the prior art, coplanarity is often measured using a laser and a 
complementary metal-oxide semiconductor (CMOS) or CCD camera connected to a 
computer via a framegrabber. This is an extremely slow process. To facilitate increased 
speed of the process the prior art has proposed alternate conveyor mechanisms to convey 
the part in the field of view of the camera in a more expeditious manner. 

The operation and application of 3-D sensors is known in the art as 
exemplified by U.S. Pat. Nos. 4,238,147, 4,590,367, and 5,028,799. The plane of light 
may be formed by a collimated laser beam spread by a cylindrical lens or by a spot of light 
which is deflected by a mirror mounted on a galvanometer, mechanical motion, spinning 
mirror, or acousto-optic deflector to form an equivalent plane made of a group of spots. 
The surface depth information is obtained by reading a signal for each part of the plane 
from a position sensor. 

Despite the solutions proposed by the prior art a significant need exists to provide 
a higher speed product to measure coplanarity. 



Summary of the Invention 

The present invention provides for both methods and apparatus relating to 
evaluating objects having three-dimensional features. 

A first aspect of the present invention provides for a method for locating three 
dimensional data in a three dimensional data set characteristic of an object wherein the 
three dimensional data is characteristic of a three dimensional feature in the object. The 
method includes acquiring a two-dimensional data set characteristic of the object wherein 
said two dimensional data set includes a plurality of values, each value including an 
address and locating those values in the two dimensional data set which are characteristic 
of the three dimensional feature in the object. A three-dimensional data set is also 
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acquired which is characteristic of the object wherein the three-dimensional data set 
includes a plurality of values, each value including an address. Values and addresses are 
selected from the three dimensional data set which are characteristic of the three 
dimensional feature as a function of the location of the data in the two dimensional data 
5 set which is characteristic of the three dimensional feature. 

According the preferred embodiment the function used to locate the three 
dimensional data characteristic of the 'three dimensional feature is an empirical 
relationship defined through a calibration process using a golden part. 

As second aspect of the present invention involves evaluating the quality of a IC 

10 package where said IC package includes a plurality of three dimensional features. The 
method of the second aspect of the present invention includes acquiring a two 
dimensional image characteristic of at least three of the three dimensional features on the 
IC package and acquiring a three dimensional image of at least three of the three 
dimensional features on the IC package. A two dimensional template representing the 

1 5 expected configuration of the three dimensional features on the IC package is provided. 
The two dimensional image is compared against the two dimensional template and 
rejecting the quality of the semiconductor if the comparison reveals that the two 
dimensional image does not include three dimensional data of the expected configuration. 
A third aspect of the present invention is directed toward a sensor for acquiring 

20 data from an obj ect including three dimensional features. The sensor preferably includes 
a CMOS device including plurality of pixels arranged in rows and columns, each column 
of pixels including an associated analog to digital converter and each column including 
an associated processor. According to the third aspect of the present invention at least 
one of said processors associated with at least one of said columns configured to acquire 

25 two dimensional data and said remaining columns configured to acquire three 
dimensional data. 



Brief Description of the Figures 



FIGURE 1 - a schematic illustration of a three-dimensional vision system 
of the present invention including a linear line laser, a sensor/processor and an integrated 
circuit package including a ball grid array according to the first embodiment of the 
present invention. 

FIGURE 2 - a schematic representation of the second preferred 
embodiment of the present invention including two linear line lasers positioned at 
different offset angles from a sensor/processor and an integrated circuit package 
including a ball grid array. 

FIGURE 3 - a schematic of the sensor and lighting arrangement 
configured to acquire two-dimensional and three-dimensional data characteristic of an 
object according to a second preferred embodiment of the present invention. 

FIGURE 4 - a schematic representation of a sensor/processor of the 
present invention including a 512X512 two-dimensional CMOS array including a 
plurality of A to D converters operable and reading out data from the columns in parallel 
and including a plurality of arithmetical project units which process the data the provide 
a linear profile. 

FIGURE 5 - an alternateVrepresentation of the operation of the 
sensor/processor of Figure 3 including a Jrfroality of A to D converters, cellular 
processors, and output shift registers where tn^ output shift registers temporarily store 
the linear profile. 

FIGURE 6 - a schematic representation of a plurality of profiles captured 

by a camera. 

FIGURE 7 - an illustration of the plurality of profiles joined together to 
provide a three-dimensional image or three-dimensional data set. 

FIGURE 8 - a representation of a plurality of profiles joined together. 

FIGURE 9 - a representation of a plurality of the profiles joined together 
in which the coplanarity of the features are measured. 



FIGURE 1 0 - a schematic representation of the preferred configuration 
of the sensor according to one aspect of the invention in which the sensor is configured 
to capture both two-dimensional data and three-dimensional data. 

FIGURE 1 1 - illustrates a laser line reflected from an object into a sensor. 
5 FIGURE 12 - illustrates problems created in the prior art because of 

geometric distortion. 

FIGURE 13 - indicates a scan path for a laser measuring a object 
including a three-dimensional feature. 

FIGURE 14 - illustrates problems in the prior art created by geometric 

1 0 distortion. 

FIGURE 1 5 - a flow chart illustration calibration of the present invention. 
FIGURE 15 1 - an illustration of a golden part. 
FIGURE 1 6 - a flow chart in which templates are created. 
FIGURE 1 7 - a flow chart according to an embodiment of the present 
15 invention in which 2D and 3D data are processed. 



Detailed Description of the Preferred Embodiment 



The present invention provides an apparatus and method for determining 
the co-planarity of objects having three dimensional features, and in particular an 
integrated circuit package (IC packages). In its preferred embodiment the invention 

20 provides an apparatus and method for determining the co-planarity of three dimensional 
features on an IC package. The present invention increases processing speed and 
throughput for the inspection of IC packages. In its preferred embodiment the present 
invention utilizes a CCD sensor which includes individual analog to digital converters and 
individual processors associated with each column of the CMOS sensor. The CMOS 

25 sensor is preferably configured to acquire both two dimensional and three dimensional 
images of an IC package. As used herein the difference between a two dimensional and 
a three dimensional image is that a three dimensional image is made up of a data set 
which includes data points defined by an address having three components, x, y, and z 
wherein a two dimensional image is made up of a data set defined by data points defined 



by an address having two components, x, and y . Viewed in terms of the present system 
a three dimensional data set includes a height or z axis component, while a two 
dimensional data set represents the appearance of an object as viewed from above. 

According to one aspect of the present invention, the two dimensional 
images acquired by the sensor may be used to pre-qualify the validity of the three 
dimensional images or data which is acquired and potentially eliminate the need to 
evaluate the three dimensional data. An example of this is that in inspecting an IC 
package it is known before hand the number and location of the three dimensional 
features to be inspected. Acquisition of two dimensional images including two 
dimensional data characteristic of three dimensional features in the object allows the 
system to confirm whether the number of features and their location on the IC package 
are proper. If they are not proper the system can reject the part. If they are proper the 
system can continue to evaluate the specific attributes of the three dimensional features. 

The present invention recognizes a problem in three dimensional vision 
not heretofore recognized, namely problems associated with geometric distortion. The 
present invention provides numerous techniques to solve geometric distortion. The 
preferred method according to this aspect of the invention uses the tw,o dimensional data 
to properly locate where in the three dimensional data actual three dimensional features 
are presented. These and other aspects of the present invention are described in more 
detail with reference to the figures in which like elements are numbered alike. 

With reference to the attached figures and in particular with reference to 
FIGURE 1 there is a schematic illustration of the first preferred embodiment of the 
present invention. FIGURE. 1 illustrates a sensor/processor 10 and a single offset laser 
12. FIGURES 2 and 3 illustrates a second preferred embodiment which uses two offset 
lasers. 

As shown laser 12 projects a laser line 16 over a plurality three 
dimensional features, or in this case balls 22 on a plane, in this case a ball grid array 20. 
Sensor/processor 10 captures images of the laser line 16 as it is reflected from the balls 
22 and processes them as will be herein described. 

As schematically illustrated^ FIGURES 4 and 5 sensor/processor 10 
includes a CMOS camera 10'. CMOS earned 10' is preferably a CMOS chip containing 



a 512 X 512 two dimensionaK^MOS array made up of pixels arranged in rows and 
columns. It is understood that thksensor could also be a charged coupled device (CCD) 
including pixels arranged in rows and columns. The CMOS sensor is made up of a grid 
*of pixels including 5 1 2 rows and 5 1 2\eolumns. Light reflected from balls 22 is captured 
by CMOS sensor 10' and the individual pixels are charged to a representative voltage 
level. As illustrated in FIGURE 4 the vAUta^e levels from each column are individually 
read out in parallel to 512 separate A/Dtamvertors 30. Each column is individually 
processed by five hundred & twelve separateVithmetical logical units (ALU) 32. Those 
of skill in the art would recognize that alternate logical components could be used such 
as look up tables (LUT) or programmed array ldgics (PAL). After processing, a linear 
profile is derived into registers at 34. \ 

Sensor/processor 10 is manufactured by International Vision Products 
(IVP) located at LinkOping, Sweden. Sensor/processor 10 is described in U.S. Patent 
Nos. 4,684,991 and 5,822,468 which are incorporated herein by reference. 
Advantageously, the IVP sensor operates at comparably high processing speeds enabling 
the system to utilize significantly more information than previously available for the 
inspection of IC packages. For example, the IVP sensor may output 3000 lines per 

second of 1024 bytes each. 

The CMOS sensor lOSportion of FIGURE 4 illustrates a single captured 
image including a profile of a ball 22\obstructed by noise. The plurality of ALU's 
determines the point within the captured hVage that represents the laser light line on the 
ball as compared to the surrounding noise. BV reading out the columns of data in parallel 
as illustrated and described, the present inventi^Weasures data and therefore coplanarity 
at a much faster rate compared to the prior a*?t\LUs 32 each select a single value or 
calculates a single value representative of a pointVn the laser line. While many options 
exist, the preferred techniques generally involves smoothing the data from each column 
and thresholding that data. The ALU then chooses thV maximum value. This maximum 
value best represents the laser linear 16 reflected fronWe BGA. 

FIGURE 5 is an alternate illustration of Sensor/processor 10. FIGURE 
9 includes a 512 X 512 array of photosensors wherein each column includes analog 
electronics 31 . The data are converted to digital values by\he analog to digital (A/D) 
converters 30, processed by a cellular processor 32 and stored in the registers 34. 
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Individual values derived from each a 5 1 2 X 1 column are temporarily stored in registers 
34. Thus the registers temporarily store the linear array characteristic of a linear laser 
line profile. After each linear profile is created, the laser and sensor arrangement, or 
alternatively the IC package, is shifted incrementally and another image is captured and 
5 processed as above. 

FIGURE 6 illustrates a collection of profiles taken at different locations 
on the ball grid array which would pass through registers 34. 

The linear profiles captured in FIGURES 6 are joined together as 
illustrated by FIGURES 7 and 8 to provide a three dimensional representation of the 
1 0 three dimensional features imaged, as illustrated a ball grid array. The adjoined profiles 
illustrated in FIGURES 7 and 8 provide the relative height of each ball from the wafer. 
Each ball is illustrated as 22 T and the wafer is illustrated as 23'. While FIGURES 7 and 
8 illustrate only four balls is understood that FIGURES 7 and 8 merely represent a IC 
package with three dimensional features, and specifically FIGURES 7 and 8 merely 
1 5 represent a larger ball grid array. Once an entire ball grid array is captured in an image 
form, such as that illustrated in FIGURES 7 and 8, a separate processor may evaluate the 
image to determine whether the balls 22 are coplanar and whether the wafer 23 is planar. 
The balls maybe evaluated for coplanarity by, for example, a least squares method or 
measurement of planes of repose. The measurement for coplanarity is schematically 
20 illustrated at FIGURE 9. 

Further, the wafer may bb^yaluated for planarity sampling points from 



wafer and setting a second degree porenomial using at least squares method. 
f 

As explained in greater detail below, once the height of each three 
dimensional feature, e.g. a ball or a lead, is known, the plane of repose can be calculated. 

25 The plane of repose is the plane upon which the package would rest if it were turned 
leads or balls down and set on a plane surface. Although in theory there could be an 
infinite number of planes of repose for a given part, for practical reasons in the preferred 
embodiment no more than two or three are selected. Two or three are preferred because 
selecting more would increase processing time significantly, and it has been found that 

30 selecting two or three is sufficient to determine whether the three dimensional features 
of a IC package are coplanar. Once the planes of repose have been selected, the distance 
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of each lead or ball from the plane can be calculated and compared to a preselected limit. 
If any of the leads or balls exceeds the limit, the part fails and this failure is noted. For 
process control purposes the mean and distribution of distance from the plane of repose 
may be noted. 

5 In addition, of interest to the IC manufacturer is the flatness of the surface 

or wafer upon which the balls are attached. If this surface deviates from flat by greater 
than a preselected amount, this information is sent to the process control system and 
optionally, the part is rejected. 

Another embodiment of a 2D/3D vision system 28 of the present 

1 0 invention is further illustrated in FIGURE 3 . An IC package including three dimensional 
features 22 is placed on surface 36. As illustrated in FIGURE 2, 3D features 22 may be 
formed by the balls of a ball grid array. Light from offset lasers 12 and 14 provide a 3D 
profile from the left of a strip 16, and a 3D profile from the right of laser line 16 as 
illustrated in FIGURE 2. Laser beams 16 are reflected off each three dimensional feature 

15 22 and captured by sensor 10. This produces a information characteristic of the height, 
or, the Z axis of the feature which is termed a 3D profile as explained above. By using 
two lasers 12, 14, one from the right of the surface 36, and one from the left of the 
surface 36, two 3D profiles are obtained, a 3D left (3DL) profile and a 3D right (3DR) 
profile. These separate 3D profiles represent the same three dimensional features from 

20 the IC package but are captured from a different angle. In particular, the 3DL and 3DR 
profiles will have different "shadow" effects where the laser cannot image the entire 
surface of the IC package. These separate 3D profiles may be used together or 
separately, or may be combined to evaluate the 3D data. Examples of how the 3D data 
could be combined include averaging the values, selecting the greater of two values or 

25 selecting the lesser of two values. It is understood that a wide variety of ways exist to 
combine the values. 

As described above, sensor 10 provides for rapid processing of data. 
With reference to FIGURE 10 it is shown how sensor 10 is configured to capture two 
separate 3D profiles from separate offset lasers 12 and 14 in addition to a distinct 2D 

30 representation of the three dimensional features on the IC package. As illustrated, sensor 
10 is configured to capture both 2D and 3D images. Those portions configured to 
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capture 3D images including 3D data are configured as described above, namely the 
processor associated with each column selects a single value which is then part of a 
profile. In the preferred embodiment, one or two columns of sensor 10 are configured 
to capture 2D data as shown at 42. The 2D images include data in a gray scale format 
5 representing the three dimensional features of the IC package as viewed from above. As 
illustrated, regions 38 and 40 of CMOS sensor 10' are configured to capture 3D data 
while region 42 is configured to capture 2D data. In the preferred embodiment 2D 
region 42 is 1 or 2 columns wide. By moving sensor 10 over surface 36 2D images of 
all 3D features 22 may be captured. 

1 0 FIGURE 3 illustrates two different illumination devices, lasers 12 and 14 

and ring lights 24 and 26. As lasers 12 and 14 illuminate the IC Package, 3D images are 
captured by regions 38 and 40 of sensor 10. Lasers are preferably thin line lasers having 
a width of 10 microns and are purchased from Lasiris, located in Montreal, Canada. 
When the 2D grayscale measurements are made lasers 12 and 14 are deactivated and ring 

1 5 lights 24 and 26 are used. Ring lights 24 and 26 include dark field and bright field 
illumination and illuminate the IC package appropriately to capture a 2D image. Ring 
lights 24 and 26 are available for purchase from Advanced Illumination in Rochester VT. 

As shown, light reflected from the IC package from both lasers 12 and 14, 
and ring lights 24 and 26 are directed through a telecentric lens system 27 and the 

20 reflected light is focused on sensor 10. Telecentric lens system 27 minimizes distortion 
by allowing light to pass in a parallel path from the IC package to sensor 10. Telecentric 
lens of the type used in the preferred embodiment are available for purchase from Sill 
Optic in Wendelstein, Germany. In the preferred embodiment ring lights 24 and 26 which 
are operative in capturing 2D images, and lasers 12 and 14 which are operative in 

25 capturing 3D images are actuated sequentially for each position of the IC package 
relative to system 28. System 28 is moved over IC package to acquire the 3D and 2D 
data described above. It is understood that for each position the sensor system is in 
relative to the IC package two images are captured, the 2D and 3D images. Typical 
operation would be as follows: lasers 12 and 14 would illuminate and sensor 10 would 

30 capture 3D data; lasers 12 and 14 would deactivate and ring lights 24 and 26 would 
activate at which time sensor 10 would acquire the 2D data; either of the sensor system 
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or IC package would be moved and the process would repeat, until all necessary data 
was acquired such that 3D and 2D images could be created characteristic of the object. 
It is understood that alternately system 28 could capture all 3D data and then go back 
and capture the 2D, or capture the 2D first followed by the 3D data. 
5 When inspecting IC packages having 3D features it is important that those 

features be coplanar. For purposes of processing the data characteristic of the 3D 
features it is convenient to think in terms of principal planes and axes. It is understood 
that the location of any feature is defined by an address, which is typically made up of an 
x, y, and z component. The principal axes are the x, y, and z axes, and the principal 

1 0 planes are the x-y , y-z, and x-z planes. Each location can be determined with respect to 
these principal axes and planes. In general, it is convenient to use the x-y plane as the 
horizontal plane for determining lead or ball locations in a 2D space, and use the z axis 
for determining the height of the tops, or bottoms, of the leads or balls with respect to 
the x-y plane; i.e. 3D space. Of course, it is well known that should the actual physical 

1 5 coordinates not coincide with these planes, simple transformations translating the data 
to use these planes and axes are available, and have been incorporated for ease of use. 
For example, if the IC packages are processed on a slight angle, the actual positions can 
be rotated with respect to the principal planes, and transformation of the data to the 
principal planes, or rotation of the principal planes, can be performed for ease of use. 

20 With reference to FIGURE 15a calibration is preferably performed at set 

up to develop the relationship between the data measurements and real world 
coordinates. The calibration establishes the relationship between the three data sets, one 
2D data set, and two 3D data sets. The calibration provides the appropriate scaling and 
correction factors that arise to convert the sensor data to the real world planar positions 

25 and heights. The calibration is performed by processing a specially machined fixture, 
called a "golden part" 50 illustrated in FIGURE 1 5', which contains specifically defined 
survey points, or fiducial marks, and step wedges that are precision machined and 
measured. The golden part 50 may be of a wide variety of configurations, including a flat 
plane, a configuration exactly representing an ideal IC package identical to the IC 

30 package to be inspected, or a planar block with steps and inclined planes. 
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With reference to FIGURE 1 5', and in the first preferred embodiment the 
golden part 50 is substantially planar and includes an etched n X" mark 52 therein. The 
features of the golden part 50 when scanned with the sensor provide the necessary 
information for computing the scaling and correction factors including a transformation 
5 between 2D and 3D space as will be described in greater detail below. 

As shown in FIGURE 15 the process is started at 100, and sensor 10 is 
initialized at 102 to acquire data in array mode. The calibration could be performed in 
line scan mode but is typically performed more quickly in array scan mode. Array mode 
describes a stationary relationship between sensor 10 and golden part 50. Images are 

1 0 captured and read out their entirety. Line mode involves moving sensor 10 over golden 
part 50 and reading the data out line by line. Lasers 12 and 14 are roughly aligned at 104 
by acquiring array mode images while adjusting the lasers lines to fall in the desired rows 
of the 2D image while the sensor to golden part distance is equal to a desired distance. 
The desired distance is typically a user set parameter. Aligning the lasers at this user set 

1 5 distance establishes a Z 0 plane, which is the plane that the 3D calculations reference and 
should fall in the middle of the range of values to be measured to provide the greatest 
dynamic range for the system. 

The first scans of golden part 50 are then performed. The center of each 
laser stripe 16 (the line of light created by the laser) from laser 12 and 14 is measured at 

20 106 and the sensor is moved at 108. The sensor system moves to all locations. The 
process of measuring the center stripes is continued at 108 and 1 10 by moving the sensor 
until the sampling is done at 110. The plane of the golden part is measured at 112. 
Measuring the plane involves taking 3D data characteristic of the planar portion of the 
golden part. The offset for the measurements of the center stripes, i.e. the distance 

25 between center points for laser stripes, are calculated at 114. Following the offset 
calculation, there is a fine tuning adjustment in the laser alignments at 1 16. The purpose 
of fine tuning is to provide a correction table for later processing. The result of slight 
misalignment of the lasers is to perceive the measured object as tilted, when it may be 
perpendicular to the optical axis of the system. By measuring an object known to be 

30 perpendicular to the optical axis, such as golden part 50, the offsets can be measured, 
stored and used to determine if during actual processing lasers 
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12 and 14 are aligned, if it not aligned the offsets can subtracted from subsequent scans 
to correct for misalignment. 

Following the measurements for golden part orientation, and the laser 
alignments, another scan is performed and the data is acquired at 118 for targets on 
5 golden part 50. The data acquired from the golden part is processed and fiducials are 
found at 120. The fiducials are survey points, or landmarks, on golden part 50 that 
determine the orientation of the part, and allow for the relative determination of positions 
of the desired target objects. This allows for a qualitative determination of the location 
of the desired target objects without having to know their absolute positions, such as 

10 occurs in the case of a an individual IC package positioned in a pocket of tray the tray 
and the IC package has shifted. The fiducials in the preferred embodiment are the lines 
of the etched "X" mark 52 and steps 54 on golden part 50. The 2D positions, the x and 
y locations, of the target data is calibrated at 122. This information is determined from 
the known positions of the precisely measured features of golden part 50, and compared 

15 with the information acquired from the 2D grayscale scan of golden part 50. An 
alternative method is to etch the fiducial marks into the fixture that holds the part and 

perform the same measurements as above. 

j 

If the golden part includes step wedges (precision vertical steps on the 
golden tray), those step wedges are found at 124. The step wedges provide the height 

20 information for calibrating the Z data at 126, or altitudes of specified 3D features from 
the 3D left and 3D right laser scans. The height information may also be obtained from 
the depth of etching of "X" mark 52 of golden part 50. The step wedge information 
provides an alternate method of determining the relationship between the displacements 
and altitudes. The step wedge is scanned and the relative altitudes of the steps are 

25 calculated. These are compared to the actual distances between steps measured by some 
other precision means and stored in the system. Calculating the relationship between the 
stored measurements and the measured values enables the sensor to be calibrated for 
distance along the Z axis relative to the 2D information. 

An alternative method of calibrating Z distances is to use a Z-axis motion 

30 control associated with the sensor system 28 shown in FIGURE 3 to move the golden 
part through the sensor's range of measurement. Golden part 50 is moved to a position 
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along the in Z axis and Z-motion encoders (not separately shown) are queried to 
establish the precision location on the Z axis for the part. Sensor 10 measures the 
location of the part and compares it to the actual location. This is repeated for several 
locations in the range of measurement of sensor 10 and the resulting relationship between 
5 measured locations and actual locations is used to calibrate the scale of Z-axis 
measurement. 

Once the Z-axis scale has been calibrated, the system finds planes at 128 
and is calibrated for rho, phi and theta at 130. Rho, phi and theta describe the attitude 
(i.e. pitch, yaw etc.) of the golden part in space in relation to the plane defined by the X, 

1 0 Y motion axes. Either sensor 10 or golden part 50 may be moved to effect measurement 
of golden part 50. In this calibration at least three non-collinear points on the golden 
part are measured and the plane that they describe in three-space is determined. The 
angles in three-space that the measured plane makes with respect to the X, Y motion of 
the scanning apparatus are stored as rho, phi and theta, along with the point in space 

1 5 selected as the center of the previously calculated Z 0 plane. This defines a relationship 
between the X, Y motion of sensor system 28 relative to the known plane of golden part 
50. These three angles and one point, along with the X, Y, and Z scales previously 
calculated and the fine calibration data from the laser angle calibration enable any two 
measured 3D points in the field of view to be adjusted to provide an accurate relative 

20 distance from the sensor. The calibration process is done at 132. 

One problem associated with 3D sensors which has heretofore not been 
recognized by the prior art is geometric distortion. Geometric distortion is an artifact 
caused by the fact that the laser sensor combination is offset relative to the 3D object 
being scanned. This problem is illustrated in FIGURES 11-14. When acquiring 3D data 

25 with an oblique laser/sensor combination, there is some geometric distortion in the 
measurements by virtue of the displacement of the reflected laser beam due to the height 
of the object scanned. In laser/sensor imaging, a laser projects a line 16 on an object. 
This line is then imaged using an array type sensor arranged at an angle, theta, to the laser 
as shown in FIGURE 1 2. Sensor 10 images a laser line 16 projected by a lasers 12 or 14, 

30 reflected off the object plane 13 and through a lens system 27. As shown in FIGURES 
1 1 and 12 if the Z displacement is zero sensor 10 images laser line 16 from object 13 in 
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the center of the sensor. FIGURE 12 shows that a Z height causes the images to be 
sensed by the sensor off its center. The 3D data is extracted as shown in FIGURE 1 2 by 
measuring the displacement d of the laser line 16 on the sensor 10 caused by interposing 
an object 13' in the path of the laser line. The displacement, d 3 is proportional to the 
5 altitude of the obj ect, Z, and the angle, theta, between the laser and the optical axis of the 
sensor system. A full 3D map of an object is created by moving either the object to be 
studied or the sensor in a direction perpendicular to the laser stripe, acquiring data along 
the way as shown in FIGURES 1 3 and 14. The relative altitude of a point measured with 
a displacement type 3D sensor is calculated from the displacement d of the reflected point 
1 0 from a zero plane. The displacement, d, of the reflected point from the zero plane is 
related to the relative altitude, Z, by the equation: 

(1) Z = d * tan(theta). 

As shown in FIGURE 14 sensor 10 attempts to measure the altitude of the object at 
position X0, but laser beam 16 is intercepted by the object 13 f at position XI with an 
1 5 altitude Z, yielding the measured displacement of d. The measure d is proportional to the 
altitude of the object 13' at the point measured, and is a measure of the displacement 
from the actual location of the object 13 f . The actual location of the point measured at 
X0 is: 

(2) X0 = Xl-d. 

20 The present invention provides a plurality of ways to correct geometric 

distortion when evaluating data in 3D space. One way is to transform the entire 3D data 
set according to the equations previously referenced. This has the effect of removing 
the geometric distortion but at the cost of processing time and sampling rate. Removing 
geometric distortion in this fashion re-samples the data non-linearly , potentially moving 

25 more than one pixel to the same location while leaving other pixels empty as can be seen 
when trying to measure position X0. In addition, noise or other spurious signals can be 
combined into the data in undesirable and unpredictable ways. The advantage of this 
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method is that calculation of the templates to be used to find the features in 3D space is 
straightforward. 

Another method for processing 3D data is to use the raw data uncorrected 
for geometric distortion and process it with similarly distorted models or templates. In 
5 this method, the feature used as a template is distorted in the same fashion as the data. 
Standard methods for locating patterns in images are then employed, such as normalized 
correlation, to locate the pattern with high precision. Once the pattern is located, it can 
be measured using a variety of methods to determine its altitude and once the altitude is 
known the planes of repose and distances may be calculated. Here, two choices exist for 

10 formation of the distorted template. In the first, a synthetic or idealized model of the 
feature, described by known mathematical equations can be created and then subjected 
to the same types of geometric distortion as the data is subjected to by the sensor. The 
second is to acquire several examples of the feature to be used to create the template and 
use a combination of the acquired examples to create the template. This has the 

1 5 advantage of including in the template higher order geometric distortions that are more 
difficult to describe mathematically and model, but that show up in the data. The 
disadvantage is that this method tends to include noise and other sources of spurious 
data. Combining several different scans of similar features is used to reduce the effect 
of spurious data. 

20 The third method of processing data and the preferred method is to 

acquire both 2D and 3D data of the same part. The 2D and 3D data are preferably 
acquired simultaneously in the invention described herein, but this is not required. In this 
method, the 2D data is processed to locate the 3D features in question to high accuracy. 
This information is then used to guide the location of the 3D features in the 3D data sets. 

25 The transformation between 3D data space and 2D data space is provided from the 
golden part calibration. An advantage of this method is that the part can be pre-checked 
in 2D space and if the part fails the 2D processing, i.e. the wrong number of features is 
found, or the features are found at the wrong location, processing can be stopped without 
performing 3D calculations, since presumably the part will be rejected based on the 2D 

30 processing. The expected altitude of the 3D feature is required by this method, since the 
geometric distortion is a function of altitude. Since the type of part being inspected is 
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known, the expected altitude of the feature can be derived from a priori knowledge of 
the part geometry, or from data acquired at the time the template was formed, if it was 
formed from actual scans of the part. Once the 2D location of the part is known and the 
expected offset of the part from the Z 0 plane is known, the 3D feature can be roughly 
5 located. The exact location of the feature in 3D may differ slightly from the expected 
location due to small variations in the altitude of a given feature. If the feature location 
differs significantly from the expected location, the part is defective and should 
presumably be rejected. 

Once the feature is located in 3D space with the assistance of 2D data, 

10 two preferred choices exist for processing. In the first, the area of the 3D image data 
containing the 3D feature can be transformed to remove the geometric distortion and the 
data processed as if it were undistorted. This has all of the advantages and disadvantages 
mentioned above, except that since a much smaller amount of data is being transformed, 
the speed disadvantage is not as great. In the second, a transformed template, or one 

1 5 derived from scanned data is used on the untransformed data to measure the feature 
directly. 

The present invention may use templates of the 3D feature to determine 
the location and height of the 3D feature. 3D data may be processed to determine the 
location of the 3D features to be measured without reference to the 2D data. Prior to 

20 processing the scanned data from the IC package to be measured, a representative IC 
package is scanned under user control. The user manually selects features to be 
measured and directs the system to construct templates for each type of feature to 
measure. A template is a mathematical representation in this case, an average 3D feature. 
These templates are stored for later use during runtime measurement. 

25 FIGURE 16 shows a flow diagram of the preferred template formation 

prdcess. At step 150, the user selects the area to be used as a template by interacting 
with a graphical user interface (GUI) which enables the user to view 3D image data and 
select a region to be made into a template. In step 152 the selected area is measured to 
locate the center of the feature within the field of view and to locate any similar features 

30 in the field of view. This is coined measuring the offset . These features are extracted 
in step 154 and filtered and combined into one average feature in step 156. This average 
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feature is stored in step 158 for use at runtime. This process is repeated until all features 
to be found at runtime have been processed as indicated at 160. It is understood that the 
processing of FIGURE 16 provides a template characteristic of an average feature. As 
described above the system may also use a "synthetic" template. 
5 Using a template of an average feature to find similar features at runtime 

has the advantage of filtering out artifact and noise that may be present in a single 
example of the feature as scanned by the sensor described herein. The alternative to 
using actual scanned data to construct a template is to use a synthetic representation to 
form the template as above described. For example, if the radius of the sphere to be 

1 0 found is known beforehand, a sphere can be formed synthetically to use as the template 
to find other spheres. A difficulty with this approach is that the sphere as scanned by the 
sensor described herein would have systematic distortions and artifacts caused by the 
geometry of the laser stripe interacting with the geometry of the part as describe above 
(geometric distortion). This will cause a sphere to appear non-spherical when scanned. 

15 Thus, in order to be used as a template, either the data to be processed has to be 
transformed to remove geometric distortions, the synthetic template has to be distorted 
to match the scan distortions, or an actual scanned feature can be used to form the 
template. All three of these methods may be used in the present invention, depending 
upon the accuracy and speed requirements of a given measurement. 

20 With reference to FIGURE 1 7 there is shown the process by which 3D 

data is acquired and evaluated for coplanarity. The scans are started at 200, and the 2D 
and 3D images are acquired by sensor 10 at 202. The images are acquired when sensor 
module 28 passes over the IC package. As described above the sensor acquires left and 
right side 3D images and 2D images. The 3D images output from sensor 10 are in the 

25 form of a profile as shown in FIGURE 4, while the 2D images are in gray scale. 

The 2D images are used as a first pass to preliminary identify and find 
defects to efficiently reject an IC package with defects in the positioning of the 3D 
features before processing for coplanarity occurs. This is an early out in finding and 
discarding defective packages. The 2D images are first pre-processed at 204. 

30 Preprocessing of 2D images typically involves filtering the images with a linear low-pass 
type filter to remove noise, such as 2D convolution with a Gaussian curve. As a part of 
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the sorting and ordering of data within an image, the data is marked according to the 
respective IC package to which it belongs. The 2D image is processed to find data 
characteristic of all features at 206, i.e. balls, leads, edges, and corners of a package. The 
finding and measurement is performed by first training an algorithm using a selected 
5 representative feature from the part to be measured. In the case of a BGA, the user 
would locate a single ball in the acquired data. The algorithm then takes this 
representative feature and uses it to train a normalized correlation routine (NCR) to 
recognize this feature. The NCR algorithm processes the 2D image and finds all the 
trained features. As is known, the found features that exceed a predetermined score are 

10 accepted as being true features. The positions of the three dimensional features in 2D 
space are determined, and the positions of edges, corners and other landmarks on the IC 
packages are determined. 

The two dimensional data characteristic of the features are sorted and 
assigned to groups at 208 of leads according to their position with respect to a given 

1 5 landmark. The groups of found 3D features in 2D space are checked for correctness at 
210, where each group of 3D features in 2D space has a specified number and location. 
The first pass on determining if an IC package is bad is made at this point. Any IC 
package with missing 3D features in 2D space, or with one or more 3D features in 2D 
space outside of predetermined tolerances for their respective positions is determined to 

20 be a defective IC package and is identified to be rejected and discarded at 212, and 
further processing of the data for a rejected package is discontinued at 214. This is an 
early removal on the checking of IC packages and from further processing of the IC 
packages determined to be defective. A defective IC package as determined from its 2D 
data does not need to have its 3D data checked, saving processing time by preventing the 

25 analysis of already determined defective IC packages. 

As is shown at 216 the 2D measurements of the 3D features are recorded. 
It is understood that these recorded measurements may be reported to the user or may 
be used for functions unrelated to the evaluation of 3D data. 

Once it is determined that the number and type of features is correct, the 

30 rough location of the features can be passed to the 3D processing routines to guide 
location of the 3D features in 3D space. The methods of locating the data characteristics 
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of 3D features as found in the 3D data set may be of many types. For example, in the 
preferred embodiment, the 2D/3D transformation derived from the golden tray 
calibration. It is also understood that the 3D features in the 3D data set may be located 
using either of the templates above-described or a prior knowledge an the geometric 
distortion equations. 

As shown in FIGURE 12 3D data is simultaneously pre-processed when 
the 2D data is pre-processed, ^re-processing of the 3D data occurs at 220. The 
preprocessing of the 3D data is prefeteibly be done in parallel with the 2D data processing 
and measurements, and then held untimeeded for further processing. The pre-processing 
of 3D data involves smoothing and fillW in the 3D data for further evaluation. This 
type of pre-processing is preferably performed using order-statistic filtering, such as a 
median filter. In addition, the data is filtered using non-linear filters that take advantage 
of the known geometry of the parts. In particular, the highest expected value in altitude 
is known from the engineering description of m«£>art. In the same fashion, the lowest 
altitude expected on the part is known to reasonable accuracy. Thus, if any process 
results in an apparent measurement being taken that exceeds either extreme in altitude, 
it can be removed without changing the measurement to be taken, since it can be assumed 
to be spurious. In the case of data where the value is greater than 1 1 0% of the predicted 
value of the height, or top, of the ball, or lead, the data i\ removed. In addition, any data 
that is below the level of the substrate by more than 10% of the substrate's predicted 
height (z value), or has a height that is less than 90% of the expected height of the 
substrate, is removed. Spurious data arises from voids found in the substrate, and from 
apparent heights resulting from the laser beam reflecting off iWore distant target as the 
scan passes over the top of a given target. Removal is importanMo prevent an abnormal 
weighting resulting from an abnormally large value, i.e. an unexpectedly large altitude, 
or depth. 

The 3D features are located at 230. The process of locating the 3D 
features in the 3D data set in the preferred embodiment utilizes the 2D/3D transform 
acquired from the golden part calibration and thus precisely locates that 3D data which 
actually represents 3D features. It is understood that other techniques may be used to 
locate the 3D features. Precise location of the feature in 3D space involves determining 
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the altitude of the parts. One way of doing this uses the template of the part as discussed 
above and fits the template to the feature data in a least-squares sense. The difference 
between the template and the data is minimized. Since the altitude of the template is 
known, the adjustment in Z required to minimize the difference between the template and 
5 the data is the difference between the template altitude and altitude of the feature under 
test. The feature can also be measured by using an arbitrary geometric shape. Many of 
the features to be measured are simple planes, therefore fitting a plane to the data in the 
least-squares sense will provide the required data. Fitting a plane gives the altitude of the 
feature with respect to a given point in space and also yields information on the attitude 

1 0 of the feature, both of which are useful information. Sub pixel resolution is possible 
using the above described template technique. When many samples are available to fit 
a shape to a feature, the location can be determined to less than a single pixel. 
Statistically, if the uncertainty of a single measurement is S, and the number of additional 
measurements is N, the certainty with which the measurement can be made increases as 

15 S/(sqrt(N)). 

The measurements of the z values are precisely determined and used to 
find the planes of repose for the leads or balls at 234. These are planes which rest upon 
the tops of all the leads. Although in theory there are a very large number of planes of 
repose, for practical reasons no more than two or three are selected. Planes of repose 

20 are calculated by ranking the measured 3D features from the tallest to the shortest. 
Beginning with the tallest features first, the highest three values are selected. These are 
fit to a plane using well-known calculations. The plane is checked and if it falls within 
a few degrees of the substrate plane it is retained. Typically, plus or minus ten degrees 
is used. The second-tallest feature is discarded and the fourth tallest feature added to the 

25 remaining two features and a plane fit as before. This plane is compared to the previous 
plane, if one was selected, and to the substrate. If the plane is within a few degrees of 
the substrate and different from the previously selected plane, it is retained. This method 
continues until the selected number of planes of repose are calculated. As stated, the 
selected number is preferably one to three. The distance of all of the features from the 

3 0 planes is then checked. If a plane exists from which no feature' s distance exceeds a pre- 
selected tolerance, the part passes. If no plane of repose is found that is closer than a 
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pre-selected distance from all of the features, the features are determined to be non- 
coplanar and the maximum value is reported 260. The measure to be used to measure 
the distance of the features from the plane can either be a Euclidean distance, where the 
distance of the feature is measured perpendicular to the plane, or a simple distance where 
5 the distance along the Z axis is used. The advantage of using the Euclidean distance is 
that it is more accurate, but more time-consuming to calculate. The advantage of using 
a simple distance measure is ease of computation at the expense of accuracy. The 
method chosen will depend upon the desired accuracy of the process. 

It is understood that the template which can be used to determine the 

10 height of the 3D feature can also be used to evaluate each 3D feature. In addition to 
comparing the heights of the leads, the shape of each 3D feature, each feature is checked 
to look for any significant defects, i.e., in the case of BGAs the quality of each ball can 
be assessed. When there is a significant deviation from the shape of the template by an 
individual lead, the lead has been formed incorrectly, such as in the case of BGAs, 

1 5 insufficient solder leaving a ball with a void space, the IC package is rejected. The defect 
can be defined in terms of shape and position. If the position of the defect is on the top 
of the ball, the height from a plane of repose could be sufficient to reject the IC package, 
or if the defect is sufficiently large, the IC package could be rejected for insufficient 
solder to affix the IC package to a substrate. 

20 In addition to checking for coplanarity of the leads, and checking the 

quality of the leads, this invention can check on warpage of the IC package. Warpage 
of the IC package is the amount of deviation from a planar shape of the package. Points 
on the IC package to be measured for warpage are included as features to be measured 
in the setup procedure. These are either flat planes, as described above, or are corners 

25 of the package, which would be measured by forming a template just as with other 
features. These feature measurements are combined in a special way once they are 
measured. The measured points are combined and a plane is fit to all of the points. If 
the distance of any substrate points exceed a predetermined tolerance, the package has 
an unsatisfactory curvature to it, and is labeled to be rejected. 
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The use of two lasers provides the advantage of revealing 3D data that 
might be obscured from view of one laser. For example, a sphere, when illuminated be 
one laser at an angle and viewed from above by the sensor would only be able to acquire 
data from one side and the top of the sphere. By illuminating the sphere from the 
5 opposite direction more of the sphere can be imaged. The question is what to do with 
two data sets taken from the same feature. If the data sets are transformed to remove 
geometric distortion prior to processing, the data sets can be combined in a non-linear 
arithmetic fashion. This method is described as non-linear because if a part of the data 
has been determined to be spurious as described above, it is not used in the combination. 

10 If both corresponding data points have been determined to be valid, they can be 
averaged. If the data sets are to be processed without having geometric distortion 
removed, they have to be processed separately. If the measurements are determined to 
be valid, as indicated by an adequate measure of quality on the least squares fit, then the 
resulting measures can be averaged. 

15 It is understood that a wide variety of hardware could be used to carry out 

the methods described above. Namely the system could save some, of an image or 
pipeline an image through its processor. 

While the invention has been described in connection with what is 
presently considered to be the most practical and preferred embodiment, it is to be 

20 understood that the invention is not to be limited to the disclosed embodiments but, on 
the contrary, is intended to cover various modifications and equivalent arrangements 
included within the spirit and scope of the appended claims, which scope is to be 
accorded the broadest interpretation so as to encompass all such modifications and 
equivalent structures as is permitted under the law. 



